feat(health): serve healthz and readyz - #72
Conversation
The scenario declared 22460 gas for a mint. Measured against the deployed binding, a mint to a receiver holding none of the token needs 69319, and one to a receiver that already holds some needs 51757. Every mint the scenario sent landed in a block with a failed status, having burned the whole limit, and trackReceipts defaults to false so the run reported each one as sent. 22460 is ERC20Noop's constant, copied. PLT-1091 covers the two scenarios that still carry it. The limit is now 75000, and the test pins it against the measurement rather than against itself. Broke the constant back to 22460 and to 200000 on purpose; the test caught both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A DeFi profile had no contract to drive. This adds a constant-product pair with the storage and gas shape of a UniswapV2 swap: both reserves, the caller's balance in each token, and an event. The contract never reverts on bookkeeping, which is the choice StorageRWv1 already makes. The balances wrap rather than check, because nothing reads them back and a load generator that fails on its own accounting stops measuring the chain. A short caller is not credited: crediting exactly what is then debited returns the slot to zero, and a zero to non-zero storage write costs four times one that changes a slot already holding a value. Under the default mix, which draws one direction, that write would land on every swap rather than the first. The reserves sit between a floor and a ceiling. Without the ceiling the input side grows without bound and the output halves every 100000 swaps, so a long run prices nothing like its start. The ceiling is also what keeps one oversized call from ending the pair: a swap of 1e49 leaves the input reserve at 1e49, and the contract has no owner and no reset. Measured, the next ordinary swap instead resets that side to the floor and pays out in full. The gas limit is 85000, read from eth_estimateGas rather than from a receipt. GasUsed is the post-refund charge and a transaction carries the pre-refund peak; sizing from a receipt put an earlier draft 20% under what its own swap needed. An account's first swap needs 79988 and every later one needs 45177, so a run in steady state declares about 44% more gas than it spends. PLT-1093 carries the prewarm change that would close that. PLT-1092 carries the chain-parameter exposure, which is the whole package rather than this constant. Every guard here was broken on purpose before it was believed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deployment has no way to tell a run that is starting from one that is stuck. The process serves /metrics and nothing else, so a probe set has nothing to gate on and a pod counts as available the moment its container starts. /healthz answers as soon as the server binds and never reads the startup sequence. /readyz refuses until the dispatcher is running. Keeping those separate is the whole point. Funding, deployment and prewarm take minutes against a cold chain. A liveness probe that reported the run dead for that window would restart the pod before it sent a transaction, then restart the next attempt at the same place, and the cause would read as a crash loop rather than a slow start. While /readyz refuses it names the phase, so a ten-minute startup shows the step it is on. Measured against the binary: healthz held 200 through a 21 second prewarm while readyz reported "prewarming accounts", then both answered once the dispatcher started. The flag and the phase are stored as one value rather than as two atomics. Two would leave a window where a reader sees the run serving while the body still names the step it left, so the status and the body would disagree about the same instant. Five mutations, five caught, including that one: split into two atomics, a reader observed a serving status carrying "funding accounts". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR SummaryLow Risk Overview
State is a single atomic pointer pairing the ready flag and phase so probe responses cannot disagree. Tests lock in the split between liveness and readiness, phase bodies, shutdown behavior, and concurrent consistency (including race-sensitive storage). Reviewed by Cursor Bugbot for commit d555a26. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Adds a small, well-tested health package serving /healthz and /readyz on the existing metrics mux, with the ready flag and phase stored as one atomic value. The separation of liveness from readiness is correct for a run with a minutes-long startup; the only gap is that readiness is dropped on the signal path only, so a duration-bounded or error exit leaves /readyz reporting running through the shutdown/flush window.
Findings: 0 blocking | 2 non-blocking | 1 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- [suggestion]
Probes.EnterandProbes.NotReadyhave identical bodies; the distinction is documentation-only. Consider havingNotReadydelegate toEnter(or drop one) so the two cannot drift apart. - 1 suggestion(s)/nit(s) flagged inline on specific lines.
| // Out of service, still alive. The run holds the pod open for the | ||
| // post-summary scrape window, and /healthz keeps answering through it so | ||
| // the kubelet does not read that hold as a hang. | ||
| probes.NotReady("shutting down") |
There was a problem hiding this comment.
[suggestion] NotReady only runs on the signal path. When --duration expires (or a background worker fails), utils.Recv returns ctx.Err() at line 396 and returns early, so this line never executes. The run then proceeds through LogFinalStats, EmitRunSummary and the PostSummaryFlushDelay sleep (25s by default) while /readyz still answers 200 running — exactly the window readiness is meant to cover, and duration-bounded runs are the common deployment shape. Registering it once for every exit after Ready() covers all paths, e.g. defer probes.NotReady("shutting down") placed right after probes.Ready() (line 371), keeping the log line where it is.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 137675a. Configure here.
#71 merged squashed, so its content reached main under a new commit and this branch's history no longer shares it. Without this merge the pull request diffs the AMM scenario again alongside the health endpoints.
NotReady ran on the signal path alone. A run whose duration expired, or whose background worker failed, returned early and never reached it, so readiness stayed true through the whole shutdown: the final statistics, the run summary, and the post-summary hold that keeps the pod open for a last scrape. That hold defaults to 25 seconds, and it is exactly the window readiness exists to cover. Deferring it right after Ready covers every path out by construction rather than by remembering to call it at each return. Measured against the same duration-bounded run, before and after. Before, /readyz answered "running" for all fourteen seconds, through the deadline at five and the ten-second hold after it. After, it flips to "shutting down" at the deadline. /healthz answers 200 throughout in both, which is what keeps the kubelet from reading a deliberate hold as a hang. The guard here is structural rather than a test: a defer at the top of the scope covers every return, and this package has no harness that drives the run's lifecycle. The measurement above is what stands in for one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Correct, and confirmed by running it rather than by reading. Fixed in
Took your suggested shape: Measured on the same duration-bounded run, before and after:
One thing worth stating plainly: the guard here is structural, not a test. A defer at the top of the scope covers every return path by construction, and this package has no harness that drives the run's lifecycle — |
#71 and #72 both merged squashed, so their content reached main under new commits and this branch's history no longer shares it. Git therefore saw the scenario files as added on both sides. Resolved toward this branch for AMM.go, AMM_test.go and ERC721.go, which carry the same contracts with their hard-coded gas constants replaced by the measured path — main holds the earlier form. Removed ERC721_test.go, which pinned a constant this branch deletes and could not compile against it. Took main's deferred NotReady in main.go. That fix landed in #72 after this branch was cut, and it covers every exit rather than the signal path alone.
The conflict resolution took main's deferred call but git had already auto-merged this branch's inline one from a region that did not conflict, so the signal path called NotReady twice. Harmless, and the opposite of what #72 did: it replaced the inline call precisely because it covered only that path.

Stacked on #71. Base is
brandon2/amm-swap-scenario, so this branch carries theAMM scenario, the ERC721 gas fix and these endpoints together, and the image it
builds is the one to deploy.
Why
sei-loadserves/metricsand nothing else. A deployment therefore has no wayto tell a run that is still starting from one that is stuck, and a pod counts as
available the moment its container starts.
That blocks the load-generator deployment in the platform repo, whose pod spec
has a
startupProbe, areadinessProbeand alivenessProbepointing at/readyzand/healthz. Against the current binary every one of them fails.What the two endpoints mean
/healthzanswers as soon as the HTTP server binds. It never reads the startupsequence.
/readyzrefuses until the dispatcher is running, and while it refuses it namesthe phase.
Keeping them separate is the point rather than a detail. Funding, deployment and
prewarm take minutes against a cold chain. A liveness probe that reported the run
dead for that window would restart the pod before it sent a transaction, then
restart the next attempt at the same place. The run would never happen, and the
cause would read as a crash loop rather than a slow start.
Measured against the binary, polling both endpoints across a real startup:
The phase in the body is there for the operator watching that window. A
ten-minute startup that answers only
503says nothing about which step is slow.One design note
The ready flag and the phase are stored as a single value, not as two atomics.
Two would leave a window where a writer has set the flag but not yet the phase,
so a reader sees the run serving while the body still names the step it left.
The status line and the body would then disagree about the same instant.
Phases
startingdeploying contractsfunding accountsprewarming accountsrunningshutting downshutting downdrops readiness while/healthzkeeps answering. The run holdsthe pod open on purpose for that scrape window, and a liveness probe that failed
during it would kill the process before its final metrics were read.
Verification
Five mutations, five caught:
NotReadymade a no-opstatus carrying
funding accountsThe fifth is worth naming. The first version of that guard passed against the
split-atomics mutation, so its failure message claimed something it could not
detect. It was rewritten to widen the window before it was believed.
gofmt,go vetandgolangci-lint runare clean. The full suite passes, andthe health package passes under
-race.🤖 Generated with Claude Code